Fix #2516: Support Packaging and Distribution #2524

liufengyun · 2017-05-24T12:54:39Z

Fix #2516: This PR enable us to run brew install lampepfl/brew/dotty and then play with dotc/dotr in command line.

The packaging is based on sbt-pack:

sbt dist/pack: generate package under dist/target/pack/
sbt dist/pack-archive: generate .zip and .tar.gz files

felixmulder · 2017-05-24T12:59:36Z

It seems to missing Dottydoc :)

Is this publishable to homebrew as is, and could you take care of that?

Otherwise LGTM

liufengyun · 2017-05-24T13:04:44Z

The scripts are assumed to support mingw and cygwin, so windows users have a chance to play with Dotty 0.1 .

liufengyun · 2017-05-24T13:08:31Z

@felixmulder I'm not sure if we need to publish the formula to Homebrew -- they have a lot of requirements on what can be accepted to brew-core, I think a shorter URL to the formula satisfy our need for now.

olafurpg · 2017-05-24T13:11:05Z

For homebrew, I recommend creating a cask so users can install dotty via brew install lampepfl/dotty/dotty. I do that for scalafmt https://github.com/olafurpg/homebrew-scalafmt

Reminds me there should really be a scala cask under scala/homebrew-scala, only someone would need to maintain it 😢

olafurpg · 2017-05-24T13:12:11Z

dist/dotty.rb

+class Dotty < Formula
+  desc "Experimental Scala Compiler"
+  homepage "http://dotty.epfl.ch/"
+  url "http://dotty.epfl.ch/dotty/dotty-0.1.1-bin-SNAPSHOT.tar.gz"


Will the artifacts always be hosted on dotty.epfl.ch? Do you have scripts to automate the deployment?

I recommend against serving the files from an EPFL server. There have examples where servers hosting artifacts have been compromised http://www.techradar.com/news/popular-video-encoding-mac-app-handbrake-compromised-with-malware I think it would be best to serve these from for example Github releases or Maven Central. I have used https://github.com/aktau/github-release to automate this, but there are other ways.

Additionally, I suspect many will write installation scripts against this url scheme. For example nix, https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/scalafmt/default.nix I would recommend to try and keep this as stable as possible.

Here is the script I used to upload artifacts to github releases and update a homebrew formula https://github.com/scalameta/scalafmt/blob/ad8559939c7d32ca78dfedf3ebc4eebc2fda7257/bin/publish.sh#L52-L71 I don't use it anymore, I only publish to Maven Central now

Thanks a lot for the tip @olafurpg , I think we should automate the publishing to Github.

olafurpg · 2017-05-24T13:12:51Z

dist/dotty.rb

+  desc "Experimental Scala Compiler"
+  homepage "http://dotty.epfl.ch/"
+  url "http://dotty.epfl.ch/dotty/dotty-0.1.1-bin-SNAPSHOT.tar.gz"
+  sha256 "4e1bda148754543844d25290a87076e6bfb0b6b0275535f97c1871e0fc5c2c4c"


If you upload the artifacts to Github releases, then the sha will be different than running sha locally. You would have to upload, download and then run sha on the dowloaded file.

olafurpg

LGTM 👍

I believe that it should be possible to provide a one-copy/paste install for all platforms in a similar way as ammonite (http://www.lihaoyi.com/Ammonite/#Ammonite-REPL) or Rust (https://www.rust-lang.org/en-US/install.html) However, I don't have much time at the moment to experiment with that. I think this is a great start!

liufengyun · 2017-05-24T14:15:09Z

dist/bin/dotd

+default_java_opts="-Xmx768m -Xms768m"
+bootcp=true
+
+PROG_NAME=dotty.tools.dottydoc.Main


@felixmulder could you give me a tip on how the dottydoc script works? The main difference from dotc is just the main method?

I run dotd examples/hello.scala locally, but it seems we need more libraries: java.lang.NoClassDefFoundError: com/vladsch/flexmark/util/options/MutableDataSet.

That should be fine - are you sure you've provided all dependencies? Dottydoc has some dependencies from Maven. See: https://github.com/lampepfl/dotty/blob/master/project/Build.scala#L304-L317

@felixmulder We've a long list of dependencies if dottydoc is included. It doesn't look very nice. How about we do some assembly for dottydoc and do it in a separate PR?

ST4-4.0.7.jar

antlr-3.5.1.jar

antlr-runtime-3.5.1.jar

autolink-0.6.0.jar

flexmark-0.11.1.jar

flexmark-ext-anchorlink-0.11.1.jar

flexmark-ext-autolink-0.11.1.jar

flexmark-ext-emoji-0.11.1.jar

flexmark-ext-gfm-strikethrough-0.11.1.jar

flexmark-ext-gfm-tables-0.11.1.jar

flexmark-ext-gfm-tasklist-0.11.1.jar

flexmark-ext-ins-0.11.1.jar

flexmark-ext-superscript-0.11.1.jar

flexmark-ext-tables-0.11.1.jar

flexmark-ext-wikilink-0.11.1.jar

flexmark-ext-yaml-front-matter-0.11.1.jar

flexmark-jira-converter-0.11.1.jar

flexmark-util-0.11.1.jar

jackson-annotations-2.2.3.jar

jackson-core-2.8.6.jar

jackson-databind-2.2.3.jar

jackson-dataformat-yaml-2.8.6.jar

jsoup-1.7.2.jar

liqp-0.6.7.jar

snakeyaml-1.17.jar

liufengyun · 2017-05-24T14:20:41Z

@olafurpg I agree, we can improve the user experience for Linux/Windows as we go. Currently sbt-pack creates zip packages, it's more visible about what's released.

DarkDimius

The idea is good, but this implementation introduces a lot of duplication and possibility for bugs. The duplication is both between scripts in this PR and already existing scripts.
I propose we merge this scripts with current scripts in ./bin/ folder.
There's a lot of nececary functionality that is implemented there, such as crash handers and restoring of tty settings. The scripts has in ./bin are inherited from @miniboxing and they are used to work on Windows(cygwin).

DarkDimius · 2017-05-24T14:54:30Z

dist/bin/dotc

@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+
+if [ -z "$PROG_HOME" ] ; then


There's a lot of copy paste and re-implementation for the current bin/dotc script.
I propose we merge the two, otherwise it will be hard to maintain and we'll be using different scripts from the ones our users do.

@DarkDimius I agree there's a potential to merge the two. But I think we can delay the merge after the Copenhagen release -- as the bin/dotc and bin/dotr have something special to the development of the compiler, such as switch between bootstrap and unbootstrap versions. We can make the decision later.

DarkDimius · 2017-05-24T14:56:02Z

dist/bin/dotd

@@ -0,0 +1,114 @@
+#!/usr/bin/env bash
+
+if [ -z "$PROG_HOME" ] ; then


No need to write separate scripts for multiple entry points. See how it's done in ./bin/dotc

@DarkDimius I didn't get your point. This piece of code is a reliable way to get the package home in order to run source "$PROG_HOME/bin/common". bin/dotr and bin/dotc do exactly the same thing.

By multiple entry points I've meant that you had similar scripts that differ only in what class they invoke in the end.

DarkDimius · 2017-05-24T14:56:12Z

dist/bin/dotr

+
+# Try to autodetect real location of the script
+
+if [ -z "$PROG_HOME" ] ; then


Same comment here.

liufengyun · 2017-05-24T16:17:31Z

@DarkDimius The new scirpts are supposed to support both mingw and cygwin, the code is from sbt-pack, I assume it's more widely tested than the code from minibox.

Actually, I found bugs related to bin/dotc about how symlink is resolved. The code from sbt-pack is more reliable.

DarkDimius · 2017-05-24T16:18:57Z

@liufengyun, ok, agreed.
But the point about duplication stands. We shouldn't have 3+ scripts to edit.

liufengyun · 2017-05-24T16:24:51Z

@DarkDimius I just removed dotd, we'll address dottydoc in another PR.

Regarding dotr and dotc, I don't see any duplication. If you mean the initial code for locating PROG_HOME, bin/dotc and bin/dotr also do the same for DOTTY_ROOT -- that piece of code cannot be shared, as sharing requires locating PROG_HOME/DOTTY_ROOT first.

liufengyun · 2017-05-25T08:07:34Z

I've published a test release in lampepfl/homebrew-brew. The standard (pre-)releases will come from @felixmulder in lampepfl/dotty after this PR is merged.

Now programmers can do the following:

Mac OS: brew install lampepfl/brew/dotty, and play with dotc and dotr.
Linux/Windows: download the zip or tar.gz, decompress and run bin/dotc and bin/dotr.

It will be great that you can try the test release and give feedback, thanks a lot.

DarkDimius · 2017-05-25T08:21:13Z

@liufengyun, just tried it on my mac. Works good so far.

DarkDimius · 2017-05-25T09:17:39Z

#2531 and #2529 contains a bunch of fixes that we should get in before release.
Otherwise we'll have hard time figuring what version has a bug that user is experiencing.

liufengyun · 2017-05-26T06:43:41Z

@DarkDimius I updated the test release with your fixes, it works well 👍

felixmulder · 2017-05-26T07:33:05Z

dist/bin/dotc

+   -nobootcp) unset bootcp && shift ;;
+      -colors) colors=true && shift ;;
+   -no-colors) unset colors && shift ;;
+      -jrebel) jrebel=true && shift ;;


Is this option being used?

felixmulder · 2017-05-26T07:33:09Z

dist/bin/dotc

+      -colors) colors=true && shift ;;
+   -no-colors) unset colors && shift ;;
+      -jrebel) jrebel=true && shift ;;
+   -no-jrebel) unset jrebel && shift ;;


@felixmulder I get them from the bin/dotc and bin/dotr. It seems it's fine to remove them. Doing it now.

liufengyun · 2017-05-26T08:06:50Z

dist/bin/dotc

-      -jrebel) jrebel=true && shift ;;
-   -no-jrebel) unset jrebel && shift ;;
-
-      -toolcp) require_arg classpath "$1" "$2" && toolcp="$2" && shift 2 ;;


We are currently not supporting toolcp and it's not properly passed in bin/dotc. So I remove it for now, and add it back when we have toolcp support.

felixmulder · 2017-05-26T08:21:52Z

Failure is due to bintray resolution?

liufengyun · 2017-05-26T08:23:55Z

Yes, I restarted it -- the sbt-gpg plugin resolution failed.

felixmulder · 2017-05-26T08:26:04Z

Could you add the artifactory resolver to the build from Fabien so that we get the caching?

liufengyun · 2017-05-26T08:42:04Z

@felixmulder The CI passes -- I didn't get the latest URL of the resolver, we do it separately?

olafurpg · 2017-05-26T08:42:56Z

FWIW, here's how I enabled the artifactory resolver only for CI scalacenter/scalafix@1c021f8#diff-fdc3abdfd754eeb24090dbd90aeec2ceR87

olafurpg · 2017-05-26T08:44:09Z

But that does not enable the artifactory for resolving plugins, I would recommend instead to enable the artifactory globally in ~/.sbt/repositories in the docker image.

felixmulder · 2017-05-26T08:57:56Z

@liufengyun - yes, it'd be good to get the patch into the docker image as well, but that's of lower priority right now.

liufengyun · 2017-05-26T09:06:35Z

Thanks @felixmulder ! If you decide to do a pre-release/release, you can follow the steps:

sbt dist/pack-archive locally
create a release on github with release message, upload the .zip and .tar.gz
update the URL and hash in the brew formula

Note: the hash is different when the artefact is uploaded. The travis CI will notice that if it's not set properly.

There's room to automate the process a little bit, but as we don't release daily, it's tolerable.

felixmulder · 2017-05-26T09:16:00Z

Let's decide at the Monday meeting what to do. Thank for the info, looks great :)

liufengyun requested review from olafurpg and felixmulder May 24, 2017 12:55

olafurpg reviewed May 24, 2017

View reviewed changes

olafurpg approved these changes May 24, 2017

View reviewed changes

liufengyun commented May 24, 2017

View reviewed changes

DarkDimius suggested changes May 24, 2017

View reviewed changes

liufengyun force-pushed the package branch from ab96496 to a308e5a Compare May 24, 2017 16:06

DarkDimius approved these changes May 24, 2017

View reviewed changes

liufengyun added 5 commits May 26, 2017 08:25

add packing and dotc/dotr

d3182f1

add brew script for install on Mac

dd9a473

fix separation in dotr

dc2f558

move brew formula to lampepfl/homebrew-dotty

0dda9e1

refine sbt task for dist

c09e87e

felixmulder approved these changes May 26, 2017

View reviewed changes

remove unsupported options

b8b6ee4

liufengyun force-pushed the package branch from adf6962 to b8b6ee4 Compare May 26, 2017 07:58

liufengyun commented May 26, 2017

View reviewed changes

felixmulder merged commit 585ec67 into scala:master May 26, 2017

liufengyun deleted the package branch May 26, 2017 08:59

Blaisorblade mentioned this pull request May 26, 2018

REPL doesn't work on Windows #4587

Closed

		@@ -0,0 +1,116 @@
		#!/usr/bin/env bash

		if [ -z "$PROG_HOME" ] ; then

		@@ -0,0 +1,114 @@
		#!/usr/bin/env bash

		if [ -z "$PROG_HOME" ] ; then


		# Try to autodetect real location of the script

		if [ -z "$PROG_HOME" ] ; then

Fix #2516: Support Packaging and Distribution #2524

Fix #2516: Support Packaging and Distribution #2524

Uh oh!

Conversation

liufengyun commented May 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixmulder commented May 24, 2017

Uh oh!

liufengyun commented May 24, 2017

Uh oh!

liufengyun commented May 24, 2017

Uh oh!

olafurpg commented May 24, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olafurpg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liufengyun commented May 24, 2017

Uh oh!

DarkDimius left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liufengyun commented May 24, 2017

Uh oh!

DarkDimius commented May 24, 2017

Uh oh!

liufengyun commented May 24, 2017

Uh oh!

liufengyun commented May 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkDimius commented May 25, 2017

Uh oh!

DarkDimius commented May 25, 2017

Uh oh!

liufengyun commented May 26, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixmulder commented May 26, 2017

Uh oh!

liufengyun commented May 26, 2017

Uh oh!

felixmulder commented May 26, 2017

Uh oh!

liufengyun commented May 24, 2017 •

edited

Loading

liufengyun commented May 25, 2017 •

edited

Loading

liufengyun commented May 26, 2017 •

edited

Loading